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ABSTRACT 



An apparatus for synchronously reproducing a video data 
and audio data includes a separating section, an audio data 
processing section, a video data processing section, and a 
synchronization processing section. The separating section 
separates a multiplexed compressed audio and video data 
into a compressed video data and a compressed audio data. 
The audio data processing section expands the compressed 
audio data to reproduce an audio data from the expanded 
audio data, and outputs the audio data to the synchronization 
processing section. The video data processing section 
expands the compressed video data in response to a control 
signal Lo reproduce a video data from the expanded video 
data, and outputs a signal indicative of each of frames, and 
a signal indicative of each of time stamps lo the synchro- 
nization processing section. The synchronization processing 
section calculates a frame rate based on the audio data 
supplied from the audio data processing section, and the 
frame signal and the time stamp signal supplied from the 
video data processing section, and outputs the control signal 
to the video data processing section based on the calculated 
frame rate without using a frame rate written in the video 
data. 

4 Claims, 5 Drawing Sheets 
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METHOD AND APPARATUS FOR 
SYNCHRONOUSLY REPRODUCING AUDIO 
DATA AND VIDEO DATA 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a method and apparatus 
for synchronously reproducing a co m pressed audio data and 
video data. 

2. Description of the Related Art 

Conventionally, as a reproduce technique of a video data 
and an audio data, there is known an audio data and video 
data reproducing system, in which the audio data and video 
data which have been digitally coded and compressed, like 
an MPEG (Moving Picture Expert Group) data are respec- 
tively expanded. In this system, the synchronization between 



15 



other to determine whether the reproduction of the video 
data proceeds more than that of the audio data. 

Also, an MPEG system decoder is disclosed in Japanese 
Laid Open Patent applications (JP-A-Heisei 8-212701 and 
JP-A-Heisei 8-214296). In this reference, reproduction 
times are set based on decoding operation limes of decoders 
(2, 3) and SCR and PTS for each register (11, 21). The PTS 
is read out from each register and transferred to a corre- 
sponding control circuit. Bit streams aTe transferred' from bit 
buffers to decode core circuits (13, 23). Each of the control 
circuit calculates the reproduction time based on the decode 
operation time of the decoder, and SCR and PTS. 

A signal processing method is described in Japanese Laid 
Open Patent application (JP-A-I leisei 6-343065). In this 
reference, a decoding system for a compressed vide signal 
and a compressed audio signal includes a local access unit 
counter for a digital AV application. An access unit header 
is generated and lost due to a storage media error or a 



the audio data and the video data (to.be .referred to^AV , v ^ansmissusn error dbeic 1 . A value called an 
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synchronization, hereinafter) is generally controlled, using 
time stamps which are written in the data. 

However, there is a case that it is not. possible. to perform 
the AV synchronization control correctly in accordance with 
the time stamps which have been written in the data, when 
a time resolution of the reproducing system is low. 

In order to avoid such a problem, the following method is 
proposed. That is, an audio reproduction lime is calculated 
from a bit rate which has been written in the audio data, and 
a reproduce byte count. The number of frames to be out- 
putted till the present time after the start of the reproduction 
is calculated based on the calculated audio reproduction time 
and the frame rate which has been written in the video data. 
'I "he calculated number of frames and the number of frames 
which have been actually outputted are compared with each 
other and the AV synchronization is controlled based on the 
comparing result. 

Also, a 24 -frame/sec data such as a movie data shown in*,. 
FIG. 1A is converted into a 60 -field/sec data of -the NXSC 
system shown in FIG. IB by the known 3:2 pull down. In 
this case, a frame rate of 30 frames/sec is written in the t 
converted data. The converted data by the 3:2 pull down has 
been processed to be reproduced in an interlace display 
mode. The converted data is converted into the -data of a 
frame structure shown in FIG. 1C, and displayed in the 
frame rate of 30 frames/sec which has been written ^n the ^ 
data, when being reproduced in the non-interlace display 
mode. 

However, when the data expanding process of a data and 
the AV synchro niza lion control are realized with a software, 

a great deal of time is spent on the video data processing, if 50 section calculates a frame rate based on the audio data 
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► unit count 

is included in each of elementary streams to suppress the 
generation of an AV sync error. The value is stored in a 
. register (238,244). An encoder (228, 230) detects the loss of 
the access unit header or an error of that to correct AV 
synchronization. 

SUMMARY OF THE INVENTION 
Therefore, an object of the present invention is to provide 
a method and apparatus for synchronously reproducing an 
audio data and video data, in-whicb AV synchronization 
control of the audio data and the video data can be per- 
formed without using a frame rate which has been written in 
the video data, when the audio data and video data which 
have been digitally compressed, like an MPEG dataj are 
expanded and reproduced, respectively. 

In order to achieve an aspect of the present invention, an 
apparatus for synchronously reproducing a -video data and 
audio data, includes a separating section, an audio data 
processing section, a video data processing section, and a 
^synchronization processing section. The separating: section 
separates a multiplexed compressed audio and video data 
into a compressed video data and a compressed audio data. 
'I "he audio data processing section expands the compressed 
audio 4a£a to reproduce an audio data from theexpanded 
;. audio data, and outputs the audio data to the sy nc brociizat i on 
rpTocessing ; -section The video data processing': section 
expands the compressed video data in response to a control 
signal to reproduce a video data from the expanded video 
data, and outputs a signal indicative of each of frames, and 
a signal indicative of each of time stamps to the synchro- 
nization processing section. The synchronization processing 



a CPU and a graphic process in the reproducing system are 
not highly efficient. Therefore, it is necessary to frequently 
perform the intermittent removal of the frames for control of 
the AV synchronization. 

In order to avoid this problem, it could be considered that 
the data of the held structure is not expanded in the 30 
frames/sec but in the original 24 frames/sec. In this case, the 
frame rate which has been written in the video data and the 
actual frame rate are different from each other. Therefore, 
the time stamps which have been written in the data must be 
used for the AV synchronization. 

In conjunction widrihe above, a video and audio data 
reproducing system is disclosed in Japanese Laid Open 
Patent application (JP-A-Hetsei 8-251543). In this 
reference, time data for a video data and an audio data are 
calculated in units of frames based on a frame update data 
and a frame rate data. The timedau are compared with each 



supplied from the audio data processing section, and the 
frame signal and the time stamp signal supplied from the 
video data processing section, and outputs the control signal 
to the video data processing section based on the calculated 
55 frame rate without using a frame rate written in the video 
data. 

In this case, the synchronization processing section 
includes a time stamp monitoring section calculating a time 
difference between the two time stamps, a frame difference 

60 counter counts a number of frames between the two time 
stamps, a calculating section calculating the frame rate from 
the number of frame difference and the time difference, and 
a control section ouiputting the control signal to the video 
'data processing section based on the calculated frame rate. 

65' In order to achieve another aspect of the present 
invention, a method of synchronously reproducing a video 
<data and^audio data, includes: 
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separating a multiplexed compressed audio and video data from the AV synchronization processing section 5. The 

into a compressed video data and a compressed audio data; frame buffer 13 stores the expanded video data. 'I rie video 

expanding the compressed audio data to reproduce an display control section 14 controls the CRT monitor 6 to 

audio data from the expanded audio data; display the expanded video data stored in the frame buffer 

a video data processing section expanding the compressed 5 ^* 

video data in response to a control signal to reproduce a audio dala processing section 4 is composed of a 

video data from the expanded video dala; * compressed audio data buffer 21, an audio decoder 22, a 

generating a signal indicative of each of frames, and a ^i buffer *** aa * M ^pro^ction control section 

signal indicative of each of time stamps; in 24 ™f compressed audio data buffer 21 stores the com- 

„ 10 pressed audio data supplied from the AV separating section 

calculating a frame rate based on the audio data, the frame 2 ^ audio decoder 22 execut£S aQalysis and aQ expanding 

signal and the time stamp signal; and process of attribute information of the compressed audio 
generating the control signal based on the calculated data, e.g., audio header information. The PCM buffer 23 
frame rate without using a frame rate written in the video stores the expanded audio data. The audio reproduction 
data. 15 control section 24 controls the speaker 7 to output the 
expanded audio data stored in PCM buffer 23. 

BRIEF DESCRIPTION OF THE DRAWINGS aw u , . c a 

The AV synchronization processing section 5 is composed 

FIGS. 1A to 1C are conceptual diagrams of a 3:2 pull of an audio dala counter 31, a clock generating section 32, 

down process; a time stamp monitoring section 33, a frame difference 

FIG. 2 is a block diagram illustrating the system configu- 20 counter 34, a frame rate generating section 35, a video frame 

ration of an apparatus for synchronously reproducing an counter 36, and an AV synchronization control section 37. 

video data and audio data according to an embodiment of the The audio data counter 31 counts a total amount of audio 

present invention; data which have been reproduced till the present time after 

FIG. 3 is a flow chart to explain the procedure of the ^ start of lhe reproduction. The clock generating section 32 

whole apparatus in the embodiment; calculates a reproduction time or an audio reference time 

. . „ . . . , using the total amount of audio data and the audio header 

FIG. 4 is a flow chart to explain the procedure of an AV Mor * matioiL Thc timc sta monitoring 33 detects 

synchronization, control; and a cunen{ ^ sump ffom me ykfco headef mformatioil and 

IK I. 5 is a diagram to explain a specific example of frame calculates a timc difference between thc current stamp and 

calculation. 30 a p rev ious time stamp. The frame difference counter 34 

nF^rRIPTIOM OF thf PRFFFRRFH counts the number of frames between the current time stamp 

DbbCKIFllON VI lHbPKblliRRED &Qd me previous time stamp m response to the detection of 

.*^j..j,i '.j..A> . ^ ^ mfe stam p Q y ^ monitoring section 33. The frame rate 

An audio data and video data synchronously reproducing generating section 35 calculates a frame rate using the time 

apparatus of the present invention will be described with " difference and the frame difference. The video frame counter 

reference to the attached drawings. 36 counts the total number of frames of video data which 

FIG. 2 is a block diagram illustrating the structure of the have been expanded and displayed from the reproduction 

audio data and video data synchronous reproduction appa- start to tne present time, as an actual frame count. 'I Tie AV 

ratus according to an embodiment of the present invention. 40 synchronization control section 37 calculates the number of 

Referring to FIG. 2, the audio data video synchronous frames of the video data which should be originally 

reproduce apparatus of the present invention is composed of expanded and displayed, as the target number of frames 

a data storage unit 1 as a data recorder, an AV separating usin g tne frame rate and the reference time. Then, the actual 

section 2, a video data processing section 3, an audio data number of frames and the target number of frames are 

processing section 4, an AV synchronization processing 45 compared such that the progress of the reproduction of the 

section 5, a CRT monitor fi and a speaker 7. ~ v i deo data to ^ c reproduction of the audio data can be 

'1 Tie data storage unit 1 stores an audio data and an video detected. Thus, the AV synchronization control section 37 

data which have been digitally coded and compressed. The set \ th f m ' mbcr ol . t0 U rcm a thin^awn 

AV separating section 2 reads out a multiplexed compressed method and sete a video data expanding method for the next 

dala from the data storage unit 1 to separates into a com- 50 l ^ * * 

pressed audio data and a compressed video data. The video decoder 12. 

data processing section 3 performs an expanding process of Nexl > lhe operation of the audio and video data synchro- 

the compressed video data supplied from the AV separating nous reproduction apparatus in this embodiment will be 

section 2. The audio data processing section 4 performs an described with reference to the flow chart of FIG. 3. 

expanding process of the compressed audio data supplied 55 In a step Al, the AV separating section 2 reads a multi- 

from the AV separating section 2. The AV synchronization plexed compressed audio data and video data from the data 

processing section 5 performs an AV synchronization con- storage unit 1 and separates into the compressed audio data 

trol. Thc CRT monitor 6 displays thc expanded video data and the compressed video data. Then, the AV separating 

and the speaker 7 reproduces the expanded audio data. section 2 stores the compressed video data in the com- 

The video data processing section 3 is composed of a 60 pressed video data buffer 11 and stores the compressed audio 

compressed video data buffer 11, a video decoder 12, a data in thc compressed audio data buffer 21. 

frame buffer 13, and a video display control section 14. The In a step A2, the expanding process and reproducing 

compressed video data buffer 11 stores the compressed process of the compressed audio data are performed. Also, 

video data supplied from the AV separating section 2. The at this timc, thc analysis of thc audio header information is 

video decoder 12 executes analysis and an expanding pro- 65 performed. 

cess of attribute information of the compressed video data, In a step A3, the AV synchronization process is performed 

e.g., video header information in response to a control signal to be later mentioned. 
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In a step A4, the expanding process and displaying 
process of the compressed video data are performed. Also, 
at this time, the analysis of the video header information is 
performed. 

In a step A5, whether or not un-processed data is left in 
data storage unit 1 is determined. When any data is left, the 
control returns to the step Al . When any data is not left, the 
control stops the reproducing process. 

Next, the AV synchronization control process (step A3) 
will be described in detail. FIG. 4 is a How chart showing the 
procedure of the AV synchronization control. 

In a step Bl, the total amount of audio data to be 
reproduced is read from the audio data counter 31. In a step 
B2, the reproduction time of the audio data as the reference 
time is calculated using the audio header information and the 
total amount of reproduced audio data, in the clock gener- 
ating section 32. 

In a step B3, the time difference is calculated from the 
current time stamp and the previous time stamp in the time 
stamp monitoring section 33. In a step U4, the frame 
difference between the current and previous time stamps is 
read out from the frame difference counter 34. 

In a step H5, in the frame rate generating section 35, a 
frame rate is calculated from the following equation (1) 
using the time difference and the frame difference: 



(frame rate)=(frame diffcicnce)/(timc difference) 



0) 



In a step B6, the target number of frames is calculated in 
the AV synchronization control section 37 using the refer- 
ence time calculated in the step B2 and the frame rate 
calculated in the step B5. After that, Lhe target number of 
frames and the actual number of frames are compared and 
the progress of the reproduction of the video data to the 
reproduction of the audio data is verified. 

Next, the number of frames to be removed is set in 
accordance with the progress of this video data, a video data 
expanding method for the next frame is set. 

Next, a method of calculating the frame rate for every key 
frame (I picture) will he described based on a specific 
example. FIG. 5 shows the specific example of the frame 
rate calculation. 

In this example, because the 60th frame is a key frame 
(I n+1 picture), the time stamp at 2500 ms of this frame is 
transmitted to the time stamp monitoring section 33. In the 
time stamp monitoring section 33, a time difference is 
determined using the current time stamp at 2500 ms and the 
previous time stamp at 2000 ms. In this example, the time 
difference is 500 ms (»2500 ms-2000 ms). Also, the frame 
difference counter 34 counts the frames detected after the 
previous key frame at the 48th frame to determine the frame 
difference. Therefore, when the I M4 . 3 picture at the 60th frame 
is detected, the frame differences is 12. At this time, the 
frame rate generating section 35 calculates the frame rate as 
follows, using the above equation (1). 

frame ratc-12 frames/500 ms-24 frames/sec. 

The frame rate can be calculated in the present invention 
as above mentioned. 

As described above, according to the present invention, 
the AV synchronization control can be correctly executed 
without using the frame rate which is written in the data, 
even when the time resolution of the reproducing apparatus 
is low. In this way, when the data subjected to 3:2 pull down 
is reproduced in the original 24 frames/sec in the non- 
interlace mode, the video data can be reproduced in syn- 



45 



chronous with the reproduction of the audio data. Thus, the 
AV synchronization can be correctly adjusted. 

Also, according to the present invention, the use of the 
calculated frame rate allows the correct AV synchronization 
control, even when the frame rate can not be normally 
acquired due to a transmission path error and a data error. 

What is claimed is: 

1. An apparatus for synchronously reproducing a video 
data and audio data, comprising: 

a separating section separating a multiplexed compressed 
audio and video data into a compressed video data and 
a compressed audio data; 

an audio data processing section expanding said com- 
pressed audio data to reproduce an audio data from the 
expanded audio data, and outputting the audio data; 

a video data processing section expanding said com- 
pressed video data in response to a control signal to 
reproduce a video data from the expanded video data, 
and outputting a signal indicative of each of frames, 
and a signal indicative of each of time stamps; and 

a synchronization processing section calculating a frame 
rate based on said audio data supplied from said audio 
data processing section, and said frame signal and said 
time stamp signal supplied from said video data pro- 
cessing section, and outputting said control signal to 
said video data processing section based on said cal- 
culated frame rate without using a frame rate written in 
said video data. 

2. An apparatus for synchronously reproducing a video 
data and audio data according to claim 1, wherein said 
synchronization processing section includes: 

a time stamp monitoring section calculating a time dif- 
ference between said two time stamps; 

a frame difference counter counting a number of frames 
between said two time stamps; 

a calculating section calculating said frame rate from the 
number of frame difference and said time difference; 
and 

a control section outputting said control signal to said 
video data processing section based on said calculated 
frame rate. 

3. A method of synchronously reproducing a video data 
and audio data, comprising: 

separating a multiplexed compressed audio and video data 
into a compressed video data and a compressed audio 
data; 

expanding said compressed audio data to reproduce an 
audio data from the expanded audio data; 

a video data processing section expanding said com- 
pressed video data in response to a control signal to 
reproduce a video data from the expanded video data; 

generating a signal indicative of each of frames, and a 
signal indicative of each of time stamps; and 

calculating a frame rate based on said audio data, said 
frame signal and said time stamp signal; 

generating said control signal based on said calculated 
frame rale without using a frame rate written in said 
video data. 

4. A method according to claim 3, wherein said calculat- 
ing includes: 

calculating a time difference between said two time 
stamps; 

counting a number of frames between said two time 
stamps; and 
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calculating said frame rate from the number of frame 
difference and said time difference. 



